test(vscuse): add local debug and Python scaffolding cases - #16476
test(vscuse): add local debug and Python scaffolding cases#16476Alive-Fish wants to merge 9 commits into
Conversation
Add semantic target adapters for the three local debug launch profiles and cover them with generated weather agent cases. The local profiles run the local lifecycle in their own preLaunchTask chain, so they require neither provision nor deploy; the Agents Playground profile serves the agent from the local machine and requires no account at all. VCB-64, VCB-65, VCB-66
Adds a pythonEnvironment semantic operation that drives the Python extension virtual environment flow, registers Python and Basic Custom Engine Agent as scaffold answers, and registers the Python remote Teams launch profile title. Weather Agent is registered for TypeScript, JavaScript, and C# only, so Python scaffolding coverage lands on Basic Custom Engine Agent instead, mapped to the three recorded Python plans.
❌ VscUse Test Plan — Tests unknownWhy these tests: Selected 6 impacted test plan(s) changed in this PR (no product code modified). Branch diff: Plans run:
ℹ️ How were these tests selected?GitHub Copilot (Claude Sonnet 4.6, high reasoning) analysed the PR title, description, and the diff between |
E2E Test Selection — AI SelectedWhy these tests: All changed files fall under vscuse test cases (rule 8 → empty), docs/specs (rule 9 → empty), or template/weather-agent changes that have no matching available e2e test cases (the only available test is a VS .NET dotnet test unrelated to weather-agent templates). Cases selected (0): Need to run more tests?Comment on this PR:
Then re-run the workflow. |
Codecov Report✅ All modified and coverable lines are covered by tests. Additional details and impacted files@@ Coverage Diff @@
## dev #16476 +/- ##
==========================================
+ Coverage 87.77% 87.84% +0.06%
==========================================
Files 647 647
Lines 34066 34216 +150
Branches 8028 8078 +50
==========================================
+ Hits 29903 30056 +153
+ Misses 2353 2343 -10
- Partials 1810 1817 +7 🚀 New features to boost your workflow:
|
…matrix Authors the weather bundle as the complete Azure OpenAI / OpenAI x TypeScript / JavaScript x remote / local Teams launch matrix, and samples the remaining surfaces once: Copilot remote, Copilot local, and Playground on both languages. Adds the (Preview) Launch Remote in Copilot (Chrome) target adapter, which unlike the declarative-agent preview requires deploy because a custom engine agent answers from a deployed bot. OpenAI cases stop at the launched surface: the run configuration declares no openai.com key and the recordings they replace type a placeholder, so only the OpenAI scaffold branch and lifecycle are honestly covered. Adds AC rows VCB-72, VCB-73, and VCB-74. Generated plans go from eighteen to twenty-five.
❌ VscUse Test Plan — Tests failureWhy these tests: Selected 15 impacted test plan(s) changed in this PR (no product code modified). Branch diff: Results: ✅ 4 passed · ❌ 11 failed (of 15 plans) Plans run:
ℹ️ How were these tests selected?GitHub Copilot (Claude Sonnet 4.6, high reasoning) analysed the PR title, description, and the diff between |
…word The browser the debug session launches keeps focus in its address bar, so the password was typed outside the sign-in form. The two recorded plans that reach this page the same way click the field first for the same reason.
The notification center keeps every notification the run has raised, so the assertion that waits for the provision or deploy success read it out of a list that also held the scaffolding, sign-in, and earlier lifecycle entries.
❌ VscUse Test Plan — Tests unknownWhy these tests: PR introduces new local-debug and playground generated plans for TS/JS weather-agent and new Python CEA scaffolding plans; selected the newly-added generated plans that directly exercise the three coverage areas added (TS local/playground, JS local, Py local and remote), validated against template changes in templates/vsc/ts/weather-agent and templates/vsc/js/weather-agent. (Also running 25 impacted test plan(s) changed in this PR.) Branch diff: Plans run:
ℹ️ How were these tests selected?GitHub Copilot (Claude Sonnet 4.6, high reasoning) analysed the PR title, description, and the diff between |
…nput box A multi-select whose one option had loaded was reported as listing none: the reader placed that option above the input box and read the 0 Selected and OK controls under the box as the whole of what the prompt lists. Where a prompt draws its rows is not a toolkit-owned invariant, so the wait now claims only that loading finished and a selectable option exists.
❌ VscUse Test Plan — Tests failureWhy these tests: PR introduces two coverage gaps: TS local debug weather agent targets (local-teams, local-copilot, playground) and new Python scaffolding for basic-custom-engine-agent (local and remote); selected the five newly-added generated plans that directly exercise those two areas. (Also running 25 impacted test plan(s) changed in this PR.) Branch diff: Results: ✅ 11 passed · ❌ 14 failed (of 25 plans) Plans run:
ℹ️ How were these tests selected?GitHub Copilot (Claude Sonnet 4.6, high reasoning) analysed the PR title, description, and the diff between |
1 participant
Extends the generated vscuse case engine with the two coverage gaps left by the initial bundle: local debug launches and Python scaffolding. Six new generated plans, each mapped 1:1 to an existing hand-recorded plan.
Local debug (
weather-agent.yml, 2 → 5 cases)Three new target adapters —
Debug in Teams (Chrome),(Preview) Debug in Copilot (Chrome), andDebug in Microsoft 365 Agents Playground.weather-ts-azure-openai-local-teamsWeather_Agent_ts_Local_Debug.jsonweather-ts-azure-openai-local-copilotWeather_Agent_ts_local_copilot.jsonweather-ts-azure-openai-playgroundWeather_Agent_Azure_OpenAI_ts_playground.jsonNew acceptance criteria:
provisionand nodeploy; the profile's ownpreLaunchTaskchain runs the local lifecycle.loginand emits no browser sign-in; the Playground serves the agent from the local machine.chatcheck under the Playground target emits the Playground message adapter.Python scaffolding (
basic-custom-engine-agent.yml, new)The Weather Agent template has no Python variant. v3
allTemplates.jsonregistersweather-agent-ts,weather-agent-js, andcustom-copilot-weather-agent-csharponly, and the v4 descriptor declares"languages": ["typescript", "javascript"]. Thetemplates/vsc/python/weather-agentfolder exists but is unregistered, Python never appears in the language picker, and the 319-plan legacy corpus contains no Python weather recording. Registering it would be a product change, so Python coverage lands on Basic Custom Engine Agent, the custom engine agent template the toolkit does offer in Python.basic-cea-py-azure-openai-remote-teamsBasic_Custom_Engine_Agent_Azure_OpenAI_py_Remote_Debug.jsonbasic-cea-py-azure-openai-local-teamsBasic_Custom_Engine_Agent_Azure_OpenAI_py_Local_Debug.jsonbasic-cea-py-azure-openai-playgroundBasic_Custom_Engine_Azure_OpenAI_py_playground.jsonNew
pythonEnvironmentsemantic operation. A Python scaffold writessrc/requirements.txtand installs nothing, and every launch profile starts the app from the workspace interpreter, so a Python case that launches without creating its environment starts an app with no dependencies. The operation is composed entirely from existing coordinate-free components:Python: Create Environment...→ filterVenv→ filter the authored interpreter → confirmSelect dependencies to installwith every dependency checked → open the notification center → assertThe following environment is selected:.Also registers the
pythonlanguage answer, thebasic-custom-engine-agentcapability answer, and theLaunch Remote (Chrome)target — the Python templates name the remote Teams launch withoutin Teams.New acceptance criteria:
pythonEnvironmentoperation drives the Venv creation flow.Launch Remote (Chrome)target reuses the Teams add-and-open transition.Verification
pnpm --dir packages/tests run test:vscuse-engine— 107 pass, 0 fail.pnpm --dir packages/tests run generate:vscuse-casesrun twice; the second printsNo generated plan changes.Follow-up
The Python local Copilot profile is titled
Debug in Copilot (Chrome)without the(Preview)prefix the TypeScript templates use. No adapter is registered for it yet, so that case is not included here.